Skip to content

fix(memory): close the rating loop and stop diluting what gets served#81

Merged
emp3thy merged 2 commits into
mainfrom
feat/memory-usefulness
Jul 22, 2026
Merged

fix(memory): close the rating loop and stop diluting what gets served#81
emp3thy merged 2 commits into
mainfrom
feat/memory-usefulness

Conversation

@emp3thy

@emp3thy emp3thy commented Jul 22, 2026

Copy link
Copy Markdown
Owner

Summary

Overnight A/B-validated fixes that raise the share of served memories an LLM marks useful (cited/shaped) from 0% (shipped config) to 22.96% (n=24 live headless sessions, 100% rating coverage). Full experiment log in autoresearch/memuse-260721-run/ on the workstation (REPORT.md / PLAN.md).

Defects fixed

  1. Rating loop never closed. The Stop hook was registered async: true, but the RATE_MEMORIES sweep replies with a decision: "block" control-flow payload that Claude Code only honours from a blocking, stdout-attached hook. Result: 30/39 recent live sessions rated zero memories. Now is_async=False, needs_stdout=True. See docs/decisions/stop-hook-must-be-sync.md. (pythonw stdout-nulling was probed and ruled out as the cause.)
  2. Retrieval ignored the task. memory.retrieve accepted only project/tech/phase/polarity and ranked purely by popularity, so every session got the same rows. Adds optional query: BM25 over title/use_cases/hints (existing reflection_fts), RRF-fused with the popularity prior. Promotes, never discards; no-match degrades to the previous order. Deliberately not required — that breaks every existing caller.
  3. Firehose default. limit_per_bucket 20 → 5. Measured: an LLM applies ~5 memories per session (5.3 ± 2.4) regardless of how many are served (42.7 mean).
  4. No demotion for dead weight. ignored ratings were write-only; 55 never-useful memories consumed 27.5% of all rated exposures. Migration 0013 adds times_ignored/last_ignored_at (backfilled from existing history, counted per distinct session). Ranking demotes only past a 10-ignored-session floor and only for memories with zero useful history.
  5. Exposure double-count. PK includes exposed_at, so re-serves added rows — same behaviour read as 16.08% raw vs 9.25% deduped on live data. All three write paths now insert at most one row per (session, kind, memory); first source wins.

Validation

arm config n useful% (distinct) useful/session
A0 shipped config 12 0.00 0.0
A1 + sync Stop only 60 9.29 1.90
A6 this branch 24 22.96 2.58

Live A/B: real claude -p sessions against sandboxed BETTER_MEMORY_HOME copies of the production DB; 12 tasks across 5 projects; genuine exposure + rating rows. Live DB never touched.

  • Full suite: 1492 passed, 22 skipped (run with the editable install temporarily repointed at this branch, then restored).
  • Test-shape updates (async/pythonw Stop, absence of query) are golden-value assertions updated to the new contract, not weakened behaviour checks.

Deploy notes

After merge: rerun python -m better_memory.cli.install_hooks, restart Claude Code. Migration 0013 applies on first connection; the backfill makes demotion effective immediately.

🤖 Generated with Claude Code

emp3thy and others added 2 commits July 21, 2026 23:16
Three defects kept memories from being marked useful. Measured with live
headless A/B sessions against sandboxed copies of a real memory DB.

1. Nothing was ever rated. The Stop hook was registered async, but its
   RATE_MEMORIES sweep replies with a `decision: "block"` payload — a
   control-flow response Claude Code only honours from a blocking hook.
   Registered async it is silently dropped, so the rating turn never runs and
   the ranking/retention feedback loop never closes. Identical task, identical
   DB: async => 44 exposed / 0 rated / 0 useful; sync => 44 / 44 / 9 (20.45%).
   On the live DB, 30 of 39 sessions in the measured month had zero ratings.
   Ruled out by probe: pythonw.exe does NOT null stdout through a pipe.
   See docs/decisions/stop-hook-must-be-sync.md.

2. Retrieval was not conditioned on the task. memory.retrieve accepted only
   project/tech/phase/polarity and ranked by a pure popularity prior
   (useful_count + 3*times_overlooked), so every caller got the same rows
   whatever they were doing, and use_cases/hints — the columns describing when
   a reflection applies — were never scored against anything. Adds an optional
   `query`: BM25 over title/use_cases/hints, RRF-fused with the prior. It
   promotes, never discards; a query matching nothing degrades exactly to the
   previous order. Not marked required — that would break every existing
   caller for a param whose absence is already safe.

3. Nothing demoted memories that keep not mattering. `ignored` was a no-op on
   the memory row, so the ranking key had no negative term: a reflection served
   142 times and useful zero times sorted level with one never served at all.
   On a live DB, 55 such memories accounted for 27.5% of every rated exposure.
   Migration 0013 adds times_ignored/last_ignored_at and backfills them from
   existing rating history, so the signal is live immediately rather than
   relearned. Demotion applies only past a 10-session floor and only to
   memories with no useful history — the best-performing memory in the live DB
   is ignored more often than it is used.

Also drops the default limit_per_bucket from 20 to 5. An LLM working one task
draws on ~5 memories however many it is handed (measured 5.3 +/- 2.4 useful
against 42.7 exposed per session); the rest is dilution and burnt context.

Counts times_ignored once per session, not once per exposure row: a memory
retrieved five times in one session that lands nowhere failed once, not five
times. This matches how _apply_one already stamps every row for a memory with
a single classification.

Test-shape updates are golden-value assertions that pinned the old async /
pythonw Stop registration and the pre-Phase-6 absence of `query`, not
behavioural checks.

Full suite: 1492 passed, 22 skipped.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e time

A session's exposures are a set: list_session_exposures groups by (kind, id)
and the rating path stamps every row for a memory with one classification.
But the PK includes exposed_at, so every re-serve of an already-exposed
memory added a new row that existed only to inflate statistics computed over
the raw table. Measured on live data the same behaviour read as 16.08%
"useful" raw and 9.25% deduplicated.

Guard all three write paths (bootstrap, reflection retrieve, semantic
retrieve) with INSERT ... WHERE NOT EXISTS on (session_id, kind, memory_id).
First exposure wins, including its source label.

Updates test_full_rating_loop, which had codified the double-count ("PK is
distinct -> 5 new rows") as expected behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Claude BugBot Analysis

Reviewed the diff (hook install sync/stdout fix, ignored-counter migration + demotion ranking, query-relevance RRF fusion, and exposure-dedup INSERT..WHERE NOT EXISTS changes) and verified SQL parameter ordering, HookSpec field ordering, FTS5 rowid linkage, and cross-file signature consistency. No genuine defects found.

No bugs were detected in this PR.

@emp3thy
emp3thy merged commit fcd771c into main Jul 22, 2026
3 checks passed
@emp3thy
emp3thy deleted the feat/memory-usefulness branch July 22, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant